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Executive Summary 



The publication of A Nation at Risk in 1983 — which decried "a rising 
tide of mediocrity" in public education that "threatens our very future 
as a Nation" — raised a national debate about the status of education in 
the United States. That it captured a widespread concern and a sensed 
need for change in education seems clear, given that, since 1980, 45 
state legislatures have mandated additional high school graduation 
requirements. But these reforms have themselves raised new concerns 
about whether the challenge of additional requirements can well serve 
the needs of disadvantaged students. 

At the request of the Subcommittee on Elementary, Secondary, and 
Vocational Education of the House Committee on Education and Labor, 
gao conducted this study to answer three questions: (1) Have the educa- 
tion reforms that raised the requirements for high school graduation 
had an effect on the achievement of educationally disadvantaged stu- 
dents? (2) Have these reforms had an effect on dropout rates for those 
students? (3) Have these reforms had an effect on their enrollment in 
vocational education? 



Some educators have noted that adding graduation requiremeats, espe- 
cially without remedial help, could simply discourage students who are 
already disadvantaged academically, (gao defined "disadvantage" as a 
score below the 35th percentile on eighth-grade standardized reading 
achievement tests.) Additional required academic courses might also 
divert disadvantaged students from vocational education courses that 
arguably could better serve both their academic and occupational devel- 
opment. Other educators have pointed out that all students, including 
disadvantaged students, are motivated to learn if they must meet the 
academic reform requirements in order to graduate. 

gao examined student data from four large school districts in four states 
where comprehensive state-level reforms have been in effect long 
enough for a class of students to experience at least 3 years of high 
school under the new requirements. The legislated reforms affecting 
these districts included, for example, more academic course require- 
ments for graduation and a passing score on an exit test. Districts serv- 
ing large numbers of urban students were chosen to ensure data on large 
numbers of disadvantaged students. Minority students were also stud- 
ied. The four districts provided computerized records for a total of 
about 61,000 students. From these records, gao developed a design that 
used two test scores (in reading and mathematics achievement) for each 
of the four school districts, which yielded eight cases fcr analysis. 
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This is the first multistat* study to empirically examine the effects of 
reform on performance, dropout rates, and enrollment patterns. These 
findings apply to the first group of students to enter high school under 
state mandated reforms in these four districts, gao's study design, 
which emphasized ectablishing the initial effects of the reforms, does 
not permit generalization to later student groups or to other districts. 



ReSllltS in Brief With regard t0 the K r i onnance of educationally disadvantaged stu- 

dents, education refonn was neither a disaster nor a boon in the four 
districts examined. Disadvantaged students appear to have improved 
performance in three of the eight cases. However, ttese improvements 
were modest, and signs of improvement did not appear in the five other 
cases. Black students showed a pattern similar to that of disadvantaged 
students. Hispanic students showed no improvements associated with 
reform; however, this result derives from only six cases, because there 
were few Hispanic students in the two remaining cases (one school dis- 
trict). Although the reforms did not bring dramatic improvements for 
disadvantaged students, they were not excluded from the gains accruing 
to the nondisadvantaged students in these districts. Indeed, at least in 
these districts the latter were no more likely than disadvantaged stu- 
dents to show signs of improvement. (See pages 21-33.) 

The effects of reform on dropout rates were mixed. Only two districts 
had so.ind data to identify dropouts. In one, the rate increased modestly 
after reform; in the other, it decreased modestly. (See pages 41- 14) 

Increased enrollment in academic courses after reform was associated 
with a decline in vocational enrollment for disadvantaged students (data 
were available for two districts). However, this decline was small on the 
average in both districts and occurred among nondisadvantaged stu- 
dents as well. Nevertheless, there are two possible concerns. First, the 
decline occurred entirely among the type of vocational education 
courses that prepare students for the labor market rather than among 
consumer or homemaking courses. Second, disadvantaged postrefc^m 
students were somewhat less likely than disadvantaged prereform stu- 
, dents to have scheduled five or more vocational courses over 3 years of 
high school. These findings suggest that at-risk students in these two 
districts may be receiving less occupational training than their pre- 
reform counterparts. (See pages 46-50.) 

The small decreases in the average number of vocational education 
courses taken by all students in these districts translated into relatively 
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substantial decreases in total vocational education enrollment. Enroll- 
ment in vocational education courses declined 7 percent in one district 
and 11 percent in the other. Such declines raise concern that the availa- 
bility of some vocational education courses and the number of voca- 
tional education teachers emploj °d may be affected. (See pages 48-50.) 



gao compared the performance, dropout rates, and enrollment patterns. 
The prereform students were the last group that entered the ninth grade 
before the additional graduation requirements were effective, and post- 
reform students were the first group that entered the ninth grade under 
the new requirements. The student groups were 1 year apart except in 
one district in which they wore 2 years apart because of a transition 
year in implementing the reforms. 



Effects on Achievement In contrast to the fear that reform efforts might neglect the special 

needs of at-risk students, educators in all four districts told gao about a 
wide range of efforts to help the; 3 students meet the higher standards. 
For example, some districts u^ed testing results to identify students 
needing tutorial help and then targeted remedial resources to them. 
Other districts added an instructional period to the school day to help 
students meet the new requirements. Others added basic skill applica- 
tions to the coverage of topics in vocational education classes. (See 
pages 19-21.) 

The three cases of improved achievement scores for disadvantaged stu- 
dents in the postreform group came from two school districts. In one 
district, both mathematics and reading scores improved; in the other dis- 
trict, mathematics but not reading scores improved. All these gains were 
modest, ranging from 5 to 8 percentile points for the median student. In 
the two other districts, neither mathematics nor reading scores 
improved after reform. (See pages 21-26.) 

Even these modest gains may not have been real improvements. Stu- 
dents may have been learning only particular test questions or testing 
skills rather than materials that are educationally meaningful, gao's 
study design did not allov/ the determination of whether the gains found 
were educationally real, as opposed to testing artifacts. However, the 
improvements may have been real. Achievement gains might have 
resulted from the redesign of curricular content and instruction prac- 
tices to better match the existing skill base identified empirically in state 
assessment or achievement tests. It may be that this type of redesign of 

6 
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curricula and instruction is more forcefully driven when test results 
have important consequences for the districts as well as for the stu- 
dents. Notably, in the two school districts that showed improvements, 
district officials had incentives to monitor schools on the results of the 
tests because they were required to provide additional services to stu- 
dents who failed to pass the state-mandated tests. (See pages 36-39.) 



Effects on Dropout Rates GA0 analyzed dropout rates for the two districts that could provide data 

distinguishing students who quit school from those who left to continue 
their education elsewhere. The small effects of reform were not consis- 
tent between these two districts but were generally consistent for 
groups within each district. The dropout rate for disadvantaged stu- 
dents worsened by 3 percent after reform in one district but improved 
by 1 percent in the other. (See pages 41-44.) 



Effects on Vocational 
Education Enrollments 



gao examined the enrollment patterns for academic and vocational edu- 
cation courses in two districts that could provide the necessary data. In 
both districts, disadvantaged students enrolled in more academic 
courses and fewer vocational education courses after reform. This 
apparent trade-off between academic and vocational education courses 
resulted in a modest decline in vocational education courses for at-risk 
students (about half a course, on the average). (See pages 45-50.) 



How should one interpret the favorable indications of academic progress 
that gao found in three <»f eight cases and the absence of strong negative 
findings in the remaining five cases? Since past evaluations have shown 
difficulty :n achieving even modest gains for disadvantaged students, 
the mixed results reported here should not be dismissed. However, 
broader conclusions require data from more school districts. If further 
studies find similar or more favoiable effects, evaluators should deter- 
mine whether the test-score gains are educationally meaningful and the 
activities or approachf s that seem to be responsible for the positive 
effects. 



Recommendations 



This report contains no recommendations. 



Agency Comments GA ° n ° l request formaI comments on a draft of this report. 



ERLC 



PageS 



1 



GAO/PEMD-89-28 Education Reform in Four School Districts 



Contents 



Executive Summary 



Chapter 1 8 

Introduction Objectives, Scope, and Methodology 8 

The School Districts We Studied 9 

Defining Educationally Disadvantaged Students 1 1 

The Data We Used 12 

Implementation Issues 14 

Study Strengths and Limitations 16 



Chapter 2 19 

The Ef feCt Of Did Districts Target Help for At-Risk Students? 20 

VA\ l on t\ cm T*(*f nrm nn Effects on At-Risk Students 21 

^aucation Keiorm on Eff ms on Black and Hispanic students 26 

otUdent Achievement Effects on Higher-Achieving Students 32 

Possible Explanations for the Findings 36 

Conclusions 39 



Chapter 3 41 

The Effect Of the Dropout Rates 41 

Reforms on Dropout Conclusions 44 

Rates for 

Disadvantaged 

Students 



Chapter 4 45 

The Effect Of Effects on At-Risk Students 46 

Educational Reform on n ffect on students Not at Risk 47 

~~ „ . vyii Discussion 48 

Enrollment in 
Vocational Education 



Appendix I: Design and Measurement Issues 51 

Appendix II: Median Achievement Scores and Net Gain 57 

Appendix III: The Effects of Reform in Rural School 61 
Districts 




8 

Pa** 6 GAO/PEMD-89-28 Education Reform m Four School District* 



Contents 



Appendix IV: Related Multistats Studies 65 
Appendix V: Msyor Contributors to This Report 68 



10 



Tables Table 1.1: Characteristics of Four School Districts 

Table 4.1 : Average Number of Courses and Change in 46 
Number of Courses for At-Risk Students Grades 9 
Through 11 

Table 4.2: Average Number of Courses and Change in 48 
Number of Courses for Students Not at Risk Grades 9 
Through 11 

Table III: At-Risk Students 57 
Table II.2: Black Students 58 
Table IJ.3: Hispanic Students 59 
Table 11.4: Students Not at Risk 60 



Figures 



Figure 1.1: Prereform and Postreform Groups Across the 


13 


Years of Test Administration in One District 




Figure 2.1: Achievement of At-Risk Students (Median 


22 


Percentile) 




Figure 2.2: Achievement of Black Students (Median 


28 


Percentile) 




Figure 2.3: Achievement of Hispanic Students (Median 


30 


Percentile) 




Figure 2.4: Achievement of Students Not at Risk (Median 


34 


Percentile) 




Figire 3.1: Dropout Rates for At-Risk Students in 


41 


Districts A and D 




Figure 3.2: Dropout Rates for Students Not at Risk in 


42 


Districts f\ and D 




Figure 3.3: Dropout Rates lor Black Student ' in Districts 


43 


AandD 




Figure 3.4: Dropout Rates for Hispanic Students in 


44 


Districts A and D 





Abbreviation 

GAO General Accounting Office 



ERIC 



Page 7 



GAO/PEMD-89-28 Education Reform in Four School District* 

9 



Chapter 1 



Introduction 



During the late 1960's and 1970's, the achievement test scores of stu- 
dents in the United States declined significantly. Serious concerns about 
this decline and the general condition of education in the United States 
were reflected in the 1983 report entitled A Nation at Risk: The Impera- 
tive for Educational Reform , produced by the National Commission on 
Excellence in Education. That report, along with several subsequent 
reports critical of U.S. education, received wide attention and helped 
fuel an education reform movement that had already begun in several 
states. Within that reform movement, 45 states have raised their gradu- 
ation requirements since 1980, and many other reforms have been initi- 
ated in an attempt to improve student achievement. 

These reforms are primarily aimed at three targets: teachers, school 
organization, and students. Reforms aimed at teachers include merit pay 
and certification examinations. Reforms aimed at school organization 
include smaller class size and lengthening the school day or year. 
Reforms aimed at students constitute a major portion of state actions 
and include legislation that has increased the number of academic 
courses required for graduation. Often, the total number of credits 
required has been increased as well. Many states have added a require- 
ment that students must pass an exit test, designated by the state, to 
graduate, while other states have implemented minimum grade-point 
averages for participation in extracurricular activities and have limited 
the number of unexcused absences for obtaining course credit. 



Objectives, Scope, and 
Methodology 



ERLC 



The Subcommittee on Elementary, Secondary, and Vocational Education 
of the House Committee on Education and Labor posed three questions 
in requesting this study: 

Have the education reforms that raised the requirements for high school 
graduation affected the achievement of educationally disadvantaged 
students? 

Have these reforms affected the dropout rates of disadvantaged 
students? 

Have these reforms affected enrollment of disadvantaged students in 
vocational education? 

As agreed with the subcommittee, we concentrated on reforms that have 
increased high school graduation requirempnts, because these reforms 
are more likely to have had direct and measurable effects upon student 
outcomes than other kinds of reform (changes in teacher licensing, for 
example). The reforms that we examined in this report, therefore, 
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include mandates that have increased the number of academic courses 
and have required students to pass exit tests to graduate. Our scope and 
methodology are described in the next four sections of this chapter. 



We chose four states to study. When we began our work in spring 1987, 
15 states had graduated a class of students under significant new high 
school course requirements, and 14 had implemented a requirement that 
students pass an exit test for high school gr Jl, ation. (Since so few 
states had substantial experience with surf , Drms, national data on 
achievement, attrition, and vocational education would not be appropri- 
ate as a measure of their effect.) We selected from states that mandated 
various combinations of reforms aimed at high school students as part 
of comprehensive state reform legislation. 

We needed to have available a number of years of computerized student 
data records from the school districts, for reasons that we will explain 
shortly. We selected districts with relatively high concentrations of edu- 
cationally disadvantaged students to ensure that we would be able to 
detect any effect among this group. 

The four districts varied in enrollment from 40,000 to over 200,000. 
Three served students living in urban and surrounding suburban areas. 
One served an exclusively urban population. One was located ?n the 
Northeast, two were in the Southeast, and one was in the Southwest. 
Most of the students in each district were black or Hispanic. The state 
legislative mandates implemented by these districts included varying 
combinations of 

• requiring exit tests for graduation, 

• increasing academic course requirements, 

• tightening attendance rules, 

• setting "no pass, no play" rules governing participation in sports and 
other activities, and 

• requiring or making available varying degrees of remedial classes for 
those in difficulty. 

We chose districts where previous local initiatives aid iiot meet or 
exceed the new state standards. Each district we studied implemented 
changes in education practices affecting high school students in order to 
comply with major state mandates. 



The School Districts 
We Studied 
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Other details of the four districts we studied are in table 1.1. As officials 
in each site requested, and as agreed with the subcommittee, we do not 
identify the school districts or their locations. 



Tablt 1.1: Characteristics of Four School Districts 



Status of state mandates 



District 



Region 



Major reform 

cohort 

affecting 

secondary 

students 



Prior school 
School district district 
response initiative 



Year 
post- 
reform 
Pre- Post- cohort 
reform reform entered 



Number of 
students 



Ethnic enrollment 



Southwest 



Passing an exit Required for 
test required 1983-84 9th- 
for graduation grade cohort 



cohort cohort grade 9 White Black Hispanic 



8,992 8,685 1984-85 19% 50% 



28% 



More English 
and 

mathematics 
credits 
required for 
graduation 



Required for 
1984-85 ytn- 
grade cohort 



Stricter 

attendance 

rules 

Grades of "D" 
no longer 
passing 

"No pass, no 
play" rule 
implemented 6 



Required for 
1984-85 9th- 
grade cohort 



Required for 
19P4-85 9th- 
grade cohort 



Required for 
1984-85 9th- 
grade cohort 



District funded 
an extra period 



Northeast 



Passing an exit 
test required 
for graduation 



Required for 
1985-86 9th- 
grade cohort 



1,258 1,462 1985-86 



10 



64 



26 



Remedial 
courses 
required for 
students failing 
sections of exit 
test 



Remediation 
required for 
1985-86 9th- 
grade cohort 



Remediation 
provided for 
earlier cohorts 



Southeast 



More 

mathematics 
and science 
credits 
required for 
graduation 



Required :>r 
1983-84 9th- 
grade cohort 



2,165 2,255 1983-84 



44 



55 



Remedial helo 
required fc iow 
achievers 



Not fully 
implemented 
until 3rd year of 
reform 



(continued) 
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Status of state mandates 



District 



Region 



Major reform 

cohort 

affecting 

secondary 

students 



School district 
response 



Prior school 

district 

initiative 



Number of 
s tudents 

Pre- Post- 
reform reform 
cohort cohort 



Year 
post- 
reform 
cohort 
entered 
grade 9 



£thnic enrollment 
Whito Black Hispanic 



Passing 4 
courses 
required to 
participate in 
extracurricular 
activitier 



Passing all 
courses 
requireo of the 
1983-84 9th- 
grade cohort to 
participate in 
extracurricular 
activities 



Southeast 



Passing an exit Required for 
♦est required 1983-84 9th- 
for graduation grade cohon 



17,481 18,568 1983-84 



23 33 



43 



Extra period 
added in 3rd 
year of reform 



fnnplemented in 
3rd year of 
reform 



Optional extra 
period 

pro dediMst 
and 2nd years 
of reform 



Relied 
minium 
passing ^rade 



Not 

implemented 



More science 
credits 
required for 
graduation 



Required for 
1933-84 9th- 
grade cohort 



Total 



29,986 30,970 



a Refor^3 were mandated for this district over a 2-year period 

b "No pass. play' rules stipulated that students must pass all courses in order to participate in extra- 
curricular activities 



c Not applicable 



Defining 
Educationally 
Disadvantaged 
Students 



The subcommittee was concerned about the effect of reform on the 
nation s most vulnerable students, those at greatest risk of school fail- 
ure. We looked for effects on students at risk on a number of dimen- 
sions, including limited English proficiency and prior school difficulty. 
Data in the four districts' records were not consistent or complete 
enough, however, for establishing many categories. 



We defined educationally disadvantaged students as those scoring at or 
below the 04th percentile on eighth-grade standardized reading tests. 1 
We examined the progress of these at-risk students as they advanced — 
or were scheduled to advance — through high school. By our definition, 

'We used the i>4th percentile to identify at-risk students because many school districts use this score 
to determine eligibility for compensatory or remedial programs. 
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at-risk students are those who are educationally disadvantaged, not nec- 
essarily economically disadvantaged— although many surely are— as 
they enter high school. We use the terms disadvantaged and at-risk 
interchangeably in this report. 



The Data We Used We tracked the egress of the same group of students as they 

progressed from eighth grade through high school in terms of their per- 
formance on reading and mathematics tests using the school district 
data files. We tracked both low-achieving, at-risk students and higher- 
achieving students (that is, students not at risk). We examined the per- 
formances of two groups, or cohorts, of at-risk students: one group that 
did not have to meet the new graduation requirements (the prereform 
group) and one that did (the postreform group). We picked the last class 
to enter high school under prereform requirements and the first one to 
enter under postreform requirements. Thus, the principal reforms 
(increased course requirements and exit exams) were in effect through- 
out the high school years for the postreform group but did not directly 
affect the prereform group. 2 ^ 

In addition to asking us to examine effects upon at-risk students and 
students not at risk, the subcommittee asked us to analyze the effects of 
education reform upon minority-group students. Thus, we compared the 
performance of prereform and postreform groups of students separately 
for each minority group. 3 

The four districts did not, of course, all implement their reforms ir the 
same year, so we did not select the same years of test data for all four 



"It is possible that some students in the prereform cohort could have been directly affected by the 
reform if they did not progress normally and thus graduated with or after the postreform cohort. We 
kept these poorer-achieving students in our analysis because removing them from the prereform 
cohort would have resulted in an unfair comparison with the postreform cohort. However, the mea- 
surable effect of reforms would be diminished as a result of some prereform students' being affected 
by reform for 1 or 2 years after entering the ninth grade In any event, we do not believe their 
inclusion can have significantly affected o«. r analyses In two districts, this irsue is not a concern, 
because students entering high school under prereform requirements were allowed to graduate under 
them, even if they did nat progress normally. In the two other districts, students who were retained 
in a grade after entering grade 9 were only a small percentage (less than 10 percent in each case) of 
the students included in our achievement analyses and, thus, would have had a relatively modest 
effect on thoso analyses Moreover, one of these latter districts had the measured improvements in 
both reading and mathematics, so any diminishment in measurable effect from retained students did 
not obscure t\e greater achievement improvements among the postreform cohort 

3 These analyses were limited by the amount of data available— that is, the existence of a sufficient 
number of students in a minority group This limitation prevented us from analyzing the etfects of 
reform upon Asian students and upon Hispanic students in one distnet 
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districts. Figure 1.1 illustrates how we went about selecting the prere- 
form and postreform groups of students in one district. The lower row 
of the figure shows that, in this district, the reforms were instituted for 
the postreform group in the 1983-84 school year, when this group was 
entering grade 9. We tracked the progress of those ninth-graders 
through what should have been their junior year of high school in the 
1985-86 school year. Within this cohort of students, we looked sepa- 
rately at the progress of both at-risk students and students not at risk. 



Figure 1.1: Prereform and Postreform 
Groups Across the Years of Test 
Administration in One District 



School Years 





Prereform 



Postreform 



Test Third Year of High School 
Reforms Instituted 
Test Grade 8 



For comparison, the prereform group we selected was the last cohort of 
students that was not directly affected by the reforms, as shown in the 
upper row of the figure. In the district illustrated, we chose those who 
were first entering grade 9 in the 1982-83 school year — 1 year earlier 
than the postreform group — and tracked them through 3 years of high 
school to the 1984-85 scnool year. Although both cohorts were students 
in the same schools in overlapping periods, the reform requirements 
applied to th? postr^iorm cohort only. Similarly, we selected cohorts 
from tho three other districts. The central criteria were that (1) the post- 
reform colic, l nad to be fully aifected by refor ns throughout the high 
school years and (2) the prereform cohort had to be the closest ac^jacent 
cohort that did not have to meet reform requirements. We adopted the 
latter criterion to keep from getting a prereform cohort that attended 
high school in a period of deferent values and culture than the postre- 
form cohort and to ensure that test data would still be available. Our 
analyses trace the educational progress of two groups of students (the 
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at-risk students and those not at risk) through two periods (prereform 
and postreform). 4 

AH four districts could provide computer records of basic demographic 
and achievement test data. All used a national achievement test (the 
Comprehensive Tests of Basic Skills in two districts and the iowa Tests 
of Fducation Development and the Stanford Achievement Test in the 
other districts) and could provide percentile rankings for students. We 
discuss the statistical comparisons we make and the reasons why we 
selected these statistics when we present the achievement results in 
chapter 2. Only two districts had detailed withdrawal data that included 
records of transfer to other schools, so the analysis of dropout rates was 
restricted to these two districts. Only two could provide computer 
records of student transcripts for our review of vocational education 
coir ^enrollments. 

In all, we traced the progress of approximately 61,000 students as they 
moved through secondary schools, some starting as fa/ back as 1982. 
The cooperation of the four school districts was essential to our work in 
assembling the large computer data files we requested. 

Our study design took account of the need to obtain implementation 
information to understand and explain results. We discussed the imple- 
mentation and results of the reforms in interviews with school district 
officials and with principals, teachers, and guidance counselors at a 
minimum of two high schools in each district. The interviews were 
designed to supplement the student data files by providing information 
on problems encountered in implementing the reforms and qualitative 
data on possible effects of reform that might not be apparent from an 
analysis of student data files. These issues have often been controversial 
In past evaluations. 



Implementation ISSUeS lne study of education reform efforts has always been a difficult under- 
taking. A fair evaluation must select a sample of appropriate projects, 
the effects examined have to be relevant to the goals of the projects, the 
reforms have to h*ve been in iriple mentation long enough to have been 
capable of having some effects upon students, and they have to have 
been strong enough for those effects to be logically attributed to the 



In one district, secondary school reforms were implemented over a 2-year period. Students who 
graduated in 1987 had to meet an exit test requirement; in 1988, graduates also had to pass addi- 
t : onal academic courses In order to look at the effects of both reqi irements, we compared the last 
prereform cohort with the second postreform cohort (classes of 1986 and 1988, respectively). 
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reforms. In the present evaluation, the interventions in the four school 
districts were typical of the comprehensive state actions that were being 
undertaken in this time period. Implementation was powerful in that the 
reforms were mandated by state law or policy and not simply advisory 
or optional for local school districts. State legislation in three districts 
required them to increase the number of courses needed to graduate. 
State mandates in two districts required students to pass an exit test to 
graduate, where no such requirement had existed before; a third district 
was ordered by the state to replace an earlier exit examination with a 
much more difficult one; the fourth district began to implement state 
requirements to provide remedial instruction to students who scored 
low on either a state basic-skills test or a standardized achievement test. 
State mandates required additional remedial courses or courses in par- 
ticular areas in all four districts. The precise extent to v/hich these 
requirements surpassed prereform requirements and the number of 
standards already reached by students are fair questions. We expect 
that most students not at risk would pass the exit tests and would not 
receive remedial services. Some of these students would have met or 
exceeded the new course requirements in order to satisfy college admis- 
sion standards. The reforms in place in the districts we studied might 
not have had a major influence on the education of these students. How- 
ever, we believe that these reforms were among the stronger reforms in 
place in large districts, and we are confident that our interviews have 
given us adequate knowledge of the degree and kind of implementation 
in the four districts. 

With regard to the relevance of the measures of effect, the heart of the 
demand for education reform came from concerns about student per- 
formance — that is, the prolonged decline of achievement test scores. 
Concerns about possible increases in dropout rates were also prominent 
in the debates of the reform period. The nature of the reforms we 
found — proficiency testing, additional course work, more academic 
courses, remedial help, and the like — suggest a concern over these out- 
comes as well. 

Another issue concerns the question of whether the reforms had been in 
place sufficiently long to be evaluated. This study includes a period of 
between 2 and 3 years (in one district, only 2 years of high school data 
were available), which seems to us a reasonable time to show an effect. 
This is usually considered an adequate period of time after an interven- 
tion has occurred for an evaluation to be reasonable. It is true that some 
ref *m programs are so complex that a longer time period may be 
needed before their outcomes can be fairly judged. This does not appear 
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to be the case in our districts; in any event, our study is likely to be the 
first of many studies by various individuals and organizations. This 
means that a body of research should become available to determine 
whether our results are replicated in other districts and in later school 
years. 

Findly, in this particular study, another implementation issue concerns 
whether the reforms were relevant to at-risk children. That is, were the 
reforms addressed to them or to higher-achieving, precollege students? 
Further, were steps taken to help at-risk students meet the new require- 
ments, or were they left largely as they had been? We address these 
issues at the beginning of chapter 2. 



Study Strengths and 
Limitations 



Strengths This is the first study to report on how the education reforms intro- 

duced in the early 1980's and later have affected the performance of a 
sample of at-risk students from multiple states and school districts. Sev- 
eral elements of the study design constitute a strong test of the effects 
of these reforms. One strength of the study is the size and geographic 
distribution of the sample: approximately 61,000 students were included 
from four large school districts in four different states. A mayor problem 
in assessing the effect of the education reform movement from aggre- 
gate national data is that any small effects of particular changes or 
packages of reforms would tend to be overwhelmed by the large number 
of students not directly affected by those changes. We avoided this 
problem and strengthened the attributional potential of our study by 
focusing on districts in states where specific reforms were implemented. 
Our focus on states with reforms implemented and the large number of 
at-risk students included in our data base were intended to allow the 
detection of even relatively minor effects of education reform. 

Our study was also designed to minimize the potential influence on our 
statistical analyses of long-term trends not related to reform. Py com- 
paring immediately accent classes (in effect, the last preref orm and 
first postreform cohorts), we largely avoided the concern that our 
before-and-after comparisons might have been affected by major differ- 
encp* between the two groups related to long-term demographic or other 
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changes. That is, using an earlier cohort as the comparison group (for 
example, one that entered high school 5 years before the postreform 
group) would have diminished the confidence in the comparability of 
the two cohorts. 

We designed our study to minimize the likelihood that differences in stu- 
dent outcomes resulted from comparing reform and nonreform groups 
composed of different kinds of people. In this study, we could not apply 
the experimental approach for ensuring comparability — that is, ran- 
domly assigning students to reform and nonreform experiences. One fea- 
sible option would have been to compare students in reform districts 
with similar students in nonreform districts, but we judged that it would 
have been difficult to closely match a small number of reform and 
lonreform school districts in terms of all the background factors that 
might affect student outcomes. In contrast, we chose to compare reform 
and nonreform students in the same district but adUacent cohorts 
because it is reasonable to assume that a cohort is similar to its ac^jacent 
cohort in terms of many background factors. 



Limitations A limitation of the study is that our design does not allow the general- 

ization of our findings to school districts nationwide. They apply only to 
the school districts we studied. As already noted, the study design we 
used emphasized establishing the effects of reform on at-risk students 
(internal validity) rather than determining the degree to which those 
effects were typical nationwide (external validity). This derives from 
the fact that our requesters' question was focused on the former, not the 
latter. 

Another limitation concerns the possible disadvantages of using ac- 
cent cohorts in the same school district as the experimental and compar- 
ison groups. We cannot exclude the possibility that reforms technically 
applicable only to the postreform cohort also affected outcomes for the 
prereform group, who would still have been in school during part cl* the 
study period. 

Some of the consequences of increased course-taking may not have been 
reflected in our data. Students might not have enrolled in the newly 
required courses until thei r junior year, and if students were retained or 
had to repeat courses, it might not be until their senior year that the 
additional requirements were met. The high school test scores we used 
might have been earned before students enrolled in the extra courses 
and this would have had a conservative influence on our findings. 
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Achievement tests were not routinely administered to 12th -grade stu- 
dents in the districts we studied. In three districts, students took 
achievement tests for the last time in their junior year; in one district, in 
their sophomore year. We used the scores from these tests because they 
were the last high school scores available. 

The achievement measure we used might not record some types of 
changes in educational attainment. The achievement tests used by the 
districts we studied are relatively low-level tests that are oriented 
toward minimum or basic skills. If the districts taught more or less 
higher-level skills such as inferential thinking, problem-solving, and 
analysis as a result of reform, the test might not reflect these changes. 
This could also have had a conservative influence on our findings. 

Because we had no practical way to check the accuracy of student 
records independently, we did not verify the data provided by our four 
districts. We did, however, examine the data for values that were obvi- 
ously incorrect and removed these few cases from the analyses. 

We did not request formal comments on our report from the school dis- 
tricts or state agencies Kvolved. Our work was conducted in accordance 
with generally accepted government auditing standards. 
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The Effect of Education Reform on 
Student Achievement 



Some people have argued that disadvantaged students — those who 
reach high school already achieving poorly — or students who are mem- 
bers of minority racial or ethnic groups might find the incre .sed require- 
ments of education reform an insuperable barrier to graduation. 
Educationally disadvantaged students might face additional obstacles to 
completing school without receiving resources and assistance to help 
them meet new, higher standards. Thus, one guess about the likely 
effects of the higher hurdles set by reform has been an expectation of 
student failure. Further, it is possible that only the students who were 
already academically prepared to meet the challenge of the additional 
requirements mandated by reform might reap educational benefits. This 
scenario says that achievement scores could rise for nondisadvantaged 
students but decline for those who are disadvantaged. 

However, many legislators and school officials have argued that the 
reforms could have their intended effect. According to this hypothesis, 
the changed legal framework of increased requirements would be trans- 
lated by teachers into higher expectations for all students, better identi- 
fication of learning gaps, and increased services to help low-achieving 
students bridge those gaps. Students might gain in achievement because 
they were more motivated and exerting more effort in order to meet the 
new requirements. Also, educationally disadvantaged students might be 
more likely to enroll in additional academic courses to meet raised grad- 
uation requirements and might benefit more from increased exposure to 
academic material as a result of reforms than higher-achieving students. 
Indeed, many higher-achieving students have, in the past, scheduled 
academic courses that met or exceeded reform requirements in order to 
meet college admissions requirements, and they did this long before 
reforms were implemented. In this view, at-risk students would be even 
more likely to benefit from reform than more academically advantaged 
students. 

Under the second hypothesis, disadvantaged students can be expected 
to benefit from reform. But under the first, they would be likely to bene- 
fit only if assistance were available to them to help them meet the 
higher requirements put in place by reform. We looked in our four dis- 
tricts for efforts to help educationally disadvantaged students in the 
new reform climate. 
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We found evidence that educationally disadvantaged students were not 
being ignored in the four districts. Educators in all four informed us of a 
wide variety of local initiatives— exceeding state or previous district 
regulations— to help students meet the higher standards. Some of these 
efforts— for example, lengthening the school day, reducing the size of 
classes, and changing the way some students were grouped for instruc- 
tion—were aimed at the way schools were organized and had the poten- 
tial of helping all students. Other efforts were aimed at curriculum and 
instructional practices and affected groups of students likely to have 
difficulties meeting the new requirements. Still others, such as increased 
remedial instruction and support services, were aimed specifically at 
individual, at-risk students. 

All four districts implemented changes in the organization of secondary 
schools to help students schedule additional courses or acquire basic 
academic skills needed to pass state tests. These efforts included, for 
example, 

• providing an official additional period in the day, even when the state 
did not fund it, to help students take all required courses; 

• offering an optional additional period in the day, at district expense, to 
help students meet new course requirements; 

• providing self-contained or cluster programs for students needing exten- 
sive help in acquiring basic skills; and 

• reducing the size of classes in targeted schools or subject areas. 

Officials in all four districts, including districts where the exit tests were 
"low-risk" (that is, passed by almost all students) also mentioned 
changes in curriculum and instructional practices that were imple- 
mented to help students pass state tests. These changes included align- 
ing curriculum to reflect skills measured on state tests and incorporating 
basic skill material within the contents covered in vocational and gen- 
eral classes. In one district, basic-skills teachers are required to develop 
individual improvement plans for low-achieving students to target 
instruction on specific skills needed to pass exit tests. 

Along with efforts aimed at school organization, curriculum, and 
instructional practices, all four districts provided remedial activities for 
students experiencing difficulties with the new requirements. These 
efforts included 

• regularly scheduled remedial help in basic skill areas; 

• voluntary tutorials during, before, and after school or on weekends; 



Did Districts Target 
Help for At-Risk 
Students? 
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• summer work-study programs; and 

• increased individual attention from counselors. 



Effects on At-Risk 
Students 



The key question for this study in comparing student achievement 
before and after the introduction of the reforms was whether there were 
benefits for at-risk or minority students as well as for students not at 
risk. Data from our four districts showed that at-risk students did not 
consistently fall behind after the reforms, as had been feared. Indeed, 
we found some test score changes showing students gaining more from 
school after reform than before, and this was true both for low-achiev- 
ing and higher-achieving students. However, these gains, overall, were 
modest. 



We looked for evidence that the reform benefited the students' achieve- 
ment in terms of a net gain in median achievement scores. 1 We assumed 
that if the reforms were beneficial, the change in postreform students' 
achievement scores would improve relative to those of prereform stu- 
dents. The measure of reform benefit, or net gain, is that the median 
postreform group score increases more or declines less between grade 8 
and high school testing than does the median prereform group score. 
Thus, we compared two cohorts of at-risk students (prereform and post- 
reform) in four cities on two achievement measures each — reading and 
mathematics — for a total of eight comparisons of achievement scores. 

Our results included both positive and negative findings. We found no 
gain associated with the reform for five of these eight outcomes, as 
shown at the bottom of figure 2.1. 2 The most negative context of these 
was in District B, where we see not only no gain but also a decline by the 
third year of high school for both the prereform and postreform groups 
in reading achievement. Further, the decline was slightly sharper for the 
postreform group. This group started grade 8 slightly worse off than the 
prereform group (at the 22nd percentile versus the 23rd) and dropped 
to the 15th percentile, thus declining 7 percentile points, or 1 more than 
their predecessors. 



! The median score is the middle score — that is, the score above which half of the students scored and 
below which the other half scored. We used medians rather than means as summary scores for the 
cohorts because percentiles are not appropriately summarized by means. Percentile ranks are not 
made up of equal measuring units For instance, the number of additional correct answers needed to 
improve a student's rank between the percentile ranks of 5 and 10 is greater than the number needed 
to uaprove between the percentile ranks of 50 and 55. Since the computation of means requires equal 
measuring units but the computation of medians does not, we chose to analyze median percentiles. 

2 The data for this and other figures appear iu tabular form in appendix II. 
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Figure 2.1: Achievement of At- Risk Students (Median Percentile)* 
Patterns of Net Gain* 
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a The graphs compare the achievement scores of the prereform and postreform cohorts at two time 
periods For districts A, B, and D, the scores were measured during the eighth grade and after 3 years 
of high school, for District C, during the eighth grade and after 2 years of high school 

b The figure shows two patterns of net gam In graphs I and II, the scores declined for the prereform 
cohort but increased for the postreform cohort In graph III, the scores declined for the prereform 
cohorts but did not change for the postreform cohort 

c The figure shows two patterns of no net gain In graphs IV, V, VII, and VIII, scores increased for both 
cohorts but gained less for the postreform cohort In graph VI, scores declined for both cohorts but 
declined slightly more for the nostreform cohort 
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How do we assess the four other cases of no net gain? In District A, the 
prereform group had a pattern of increased performance scores from 
grade 8 to the third year of high school. In District A, the postreform 
group also improved its percentile standing in both reading and mathe- 
matics relative to the norms. Furthermore, the postreform group had 
notably higher test scores after 3 years of high school than the prere- 
form group. But the growth rate of the postreform group in high school 
was lower than tl \t of the prereform group. (This finding is reflected by 
the somewhat steeper lines shown in the graph for the prereform group 
in District A.) Moreover, the postreform group had a much higher initial 
gi au<? 8 performance than the prereform group. These higher grade 8 
scores might have resulted from changes in the community, in testing 
practices, or in attrition or might have represented stronger perform- 
ance at the elementary grades. 3 In any case, the postreform group's ini- 
tial advantage (defined as the gap between the prereform and 
postreform groups) faded somewhat in high school, despite reform 
efforts. 

We found two other examples of slightly lower rates of gain after 
reform. In District D, the postreform group started slightly ahead of the 
prereform group in both mathematics and reading but lost these advan- 
tages and ended up only equal to, or slightly lower than, its predecessor 
by the final testing year. 

With regard to our positive findings, the upper part of figur e 2.1 shows 
three clear examples of positive change in the performance of educa- 
tionally disadvantaged students. Note that in all three cases, the postre- 
form groups 1 median percentile scores increased or remained stable 
during high school, while the prereform group declined. In short, in 



4 The consistently higher scores for the postreform cohort in District A could have resulted from 
either biased attrition by low-achievers or initial differences between the prereform and postreform 
cohorts. This pattern could occur if low-achieving postreform students were less likely to take the 
high school test (whether from increased dropping out, changed district testing policy, or some other 
reason). If this were true, our analysis (which excludes students who did not take the high school 
test) would show initial and later higher scores for the postreform cohort because some low-achieving 
students would be systematically excluded from this cohort This hypothesis is more plausible 
because drop-out rates increased slightly in District A, which could indicate lai ger numbers of low- 
scorers were removed from the postreform cohort. We conducted special analyses to examine this 
issue and conclude that at-risk, postreform students' higher scores appear to have resulted primarily 
from initial differences rather than differential dropout. Part of the greater-than-expected initial dif- 
ferences result from the reform transition year that separated the cohorts in this particular district. 
At-risk students' scores in the transition year fell between those of students in the preform and post- 
reform, suggesting a trend of pre-high-school achievement improvements that widened the achieve- 
ment gap between nona4)acent cohorts. Moreover, a comparison of all grade 8 test-takers (who may 
or may not have taken the high school test) reduced the gap in initial scores between the prereform 
and postreform cohort by less than half, indicating tnat differential dropout is only a partial explana- 
tion of the higher scores among the at-risk students in the postreform cohort. 
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these three cases, the postreform group of disadvantaged students both 
improved or maintained their performance relative to the national 
norms ■ nd improved relative to the prereform group (which, in fact, 
declined). These three positive effects were found in District C in both 
reading and mathematics and in District B in mathematics. All three 
positive effects, however, were relatively modest, since the differences 
in median changes between prereform and postreform groups ranged 
from 5 to 8 percentile points (see appendix II). 

We did not do tests of significance on our results because we analyzed 
differences between all test-takers in the districts rather than between 
samples of test-takers. Moreover, given the large number of students we 
analyzed, tests of significance at conventional levels of significance 
might be likely to indicate that even very small or negligible changes 
were statistically significant, 

Oar findings did not appear to be dependent upon year-to-year fluctua- 
tion in test scores or long-term improvements in the districts. 4 Long-term 
district improvement would have been suggested if we found a trend of 
improving performarce among either cohorts. Although we did not 
have data on earlier cohorts, the fact that the scores declined for the 
prereform cohort but increased or remained stable for „he postreform 
cohort, except for reading scores in District B, suggests that the 
improvement broke with, rather than continued, a long-term ti .nd in the 
district. 

We also checked our data to see if differences in achievement may have 
resulted from age differences between the cohorts. If the postreform 
cohort consisted of more students who were overage for their grade 
placement than in the prereform cohort, we would expect the postre- 
form cohort to perform better on achievement tests, because norms for 
these tests are based on grade norms rather than age norms. We found 
that the peril 'mance of overage students in the preform and postreform 
cohorts diff d Ly less than 1 percent in all four districts.* 



4 In districts A, B, and D, in which we could inspect the net gains in the second high school year, our 
findings 01 net gain are essential'y consistent with those for the third year of high school that we 
have reported. Our conclusion about patterns of net gain or no net gain would change in only one of 
the six comparisons, and this one exception — a change from : . weak pattern of no net gain (minus 1 ) 
to a weak pattern of net gain (1) — suggests a relatively small year-to-year random fluctuation in tesi 
scores Jiat could not explain the sizable aet gains we observed m our three cases of positive effects 

5 In District A, the percentages of overage students in both cohorts was 9 1 percent In District B, 10.6 
percent of the prereform and 1 1 .4 percent of the postreform cohort were overage. In District C, the 
percentages of overage students in the prereform and postreform cohorts were 21 7 and 22.4. In 
District D, the percentages were 17.1 and 17 8. 
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These achievement results indicated that at-risk students, on the aver- 
age, definitely have not suffered "disasters" as a result of education 
reform in these four cities, and in some instances they seem to have 
prospered. But our findings on the eight outcomes were both mixed and 
modest. 



Effects on Black and 
Hispanic Students 



We paid special attention in our study to the effects of education 
reforms on minority students. We move now to the results for all black 
and Hispanic students, not just students with lower grade 8 test scores. 

For blacks, the results were about the same as we reported for students 
educationally at risk: three cases of overall net gain, five cases of no net 
gain. (See figure 2.2 on page 28.) The only difference was for reading in 
District D. Neither blacks nor disadvantaged students appeared to bene- 
fit from reform in District D, but postreform disadvantaged students' 
reading scores improved between grade 8 and later testing (see figure 
2.1), whereas postreform black students' scores declined. Figure 2.2 
thus shows two cases of greater decline after reform for black students 
(graphs VI and VIII), compared to only one for low-achieving students. 6 



^District \ nostreforni black students had higher scores at grade 8 and after 3 years of high school 
than pre. _ 4 orm black students (the good news) but their relative advantage faded during high school 
(the bad news). 
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For Hispanics, we had data for only six comparisons instead of eight, 
because of the very small Hispanic student population in District C. Our 
results were all negative: they showed no instances of net gains for the 
postreform group and two cases of greater decline after reform than 
before. Figure 2.3 on page 30 shows the reading decline for Hispanics in 
districts B and D, which worsened slightly after reform. Two other 
results — mathematics in districts B and D — show that the Hispanic stu- 
dents' performance relative to national nonrs declined somewhat during 
high school for both the prereform and postreform groups. The postre- 
form Hispanics in District A did improve their reading and mathematics 
scores over time, but after having had much higher initial levels at 
grade 8 compared to prereform students, they followed this by making 
no net gains in the rate of improvement. This was the same pattern that 
we had already seen for blacks and at-risk students in District A. 
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Figure 2.2: Achi^-r.^nt of Black Students (Median Percentile) 1 
Patterns of Nat Gain" 
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a The graphs compare the achiever "*nt scores of the prereform and postref-rm cohorts at two time 
periods For districts A, B, and D, the scores were measured during the eighth grade and after 3 years 
of high school, for District C, during the eighth grade and after 2 years of high school 

b The figure shows one pattern of net gain In graphs I, II, and III, scores declined p'C.erorm cohort 
but increased or remained the same for the postreform cohort 

c The figure shows three patterns of no net gam In graphs IV and VII, scores increased for both cohorts 
but gained less for the postreform cohort In graph V, scores remained almost unchanged before and 
after reform In graphs VI and VIII, scores declined for both cohorts but declined slightly more for the 
postreform cohort 
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Figure 2.3: Achievement of Hispanic Students (Median Percentile)* 
Pattern* of Net Gain - No Instances 
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a The graphs compare the achievement scores of the prereform and p Preform cohorts at two time 
periods The scores were measured during the eighth grade and af;er 3 years of high school 

b The figure shows three patterns of no net gain In graphs I and IV scores increased for both cohorts 
but gained less for the postreform cohort In graphs II and V, scores declined equally bof^re and after 
reforms In graphs III and VI, scores declined for both cohorts but decked slightly more for the postre- 
form cohort 

In short, our finding from these three districts is that the education 
reform experience has not helped Hispanic students. Our results for 
these groups were all negative. (Since there were few Hispanic students 
in the otherwise successful District C, we cannot tell what their per- 
formance might have been there or how it might have changed this 
overall showing). 
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Effects on Higher- 
Achieving Students 



We found that the better students did about the same as disadvantaged 
students in benefiting from reform. (See figure 2.4 on page 34.) That is, 
the students not at risk also had three instances of net gain out of eight 
comparisons. District C again showed net gains in both reading and 
mathematics. The District A mathematics results were positive in that 
the prereform group's decline improved slightly after reform. 

In terms of score changes among postreform students, our results sug- 
gest an apparently weaker performance by better students than by dis- 
advantaged students, but this observation must be interpreted with 
caution. There are four instances of no net gain in which the scores for 
the students not at risk in the postreform group declined (reading and 
mathematics in District B and reading and mathematics in District D) 
compared to only one decline for at-risk students (figure 2.1). When 
groups have been selected for extreme scores— as here for at-risk stu- 
dents and, to a lesser extent, students not at risk— there is a statistical 
tendency for the scores of the higher group to decrease and the scores of 
the 'ower group to increase upon retesting. 7 For this reason, the larger 
number of declining scores among better students compared to disad- 
vantaged scudents could reflect either a real difference or only a statisti- 
cal artifact. 



We also traced the experience of higher-achieving students not at risk: 
that is, those scoring at and above the 35th percentile in grade 8 read- 
ing. We used these results as anotner yardstick against which to mea- 
sure the results for low-achieving students and for minorities, but they 
are, of course, interesting in and of themselves. 



This statistical tendency is known as "regression toward the mean " It is related here to the impreci- 
sion of the achievement testing process that results in measurement error The more extreme the 
score, the more likely it is to include lprger errors. Thus, a very low scorer is more likely to have 

arger error from the true score in the negative direction, whereas a very high scorer is likely to have 
larger error in the positive direction Since measurement error has a random component, the low 
scorers would he expected to .mprove on average upon retesting, and the high scorers decline some- 
what on average Thus, measured scores would change without changes in t.-ue scores 
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In summary, we found that education reform in these four school dis- 
tricts did not produce more favorable results for nondisadvantaged stu- 
dents than for disadvantaged students. How can we explain these 
results? Since we had included students at and above the 35th percentile 
in our category of nondisadvantaged students, the group incorporated 
students presenting a fairly wide range of performance, and it seemed 
possible to us that this wide range might have obscured favorable 
results occurring within smaller subsets of this nondisadvantaged 
group. Therefore, we examined the group that scored between the 35th 
and 50th percentiles separately. However, we found the results gener- 
ally to be the same. 

These findings for students not at risk can be viewed as a yardstick for 
evaluating our results for at-risk students. That is, these findings show 
that the reforms have similar rather than differential effects for both 
groups. Of course, these results are limited, and we would want to see if 
data from other districts support this conclusion. 
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a The graphs compare the achievement scores of the prereform and postreform cohorts at two time 
periods. For districts A, B, and D, the scores were measured during the eighth grade and after 3 years 
of high school 

^The figure shows three patterns of net gain In graph I, scores increased for both cohorts but gained 
more for the postreform cohort In graph II, scores declined for the prereform cohort but increased for 
the postreform cohort In graph ill, scores declined for both cohorts but declined less for the postreform 
cohort 

c The figure shows three patterns of no net gain In graph IV, scores increased for both cohorts but 
gained less for the postreform cohort In graphs V, VII, and VIII, scores declined 'or both cohorts but 
declined slightly more for the postreform cohort In graph VI, scores declined equally before and after 
reform 
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Possible Explanations 
for the Findings 



The major strength of our design was its ability to detect broad changes 
in the performance of large cohorts of students. To help explain the 
trends and differences we found among the four school districts, we 
interviewed selected school officials and teachers in each district. The 
purpose of the interviews was to explore the nature of the reforms, their 
implementation, and possible explanations for our findings in the analy- 
ses of test scores for the school district. 



Districts B and C 



One finding of particular interest is the encouraging results for at-risk 
and black students in districts B and C, which we reported in figures 2.1 
and 2.2. It is not clear which reform element (or combination of ele- 
ments) might have been most important in achieving the benefit for at- 
risk students in these two districts. In District C, the reforms imple- 
mented were largely raised graduation requirements calling for addi- 
tional mathematics and science credits and a requirement for passing 
grades to participate in extracurricular activities. District C also 
required students to take a competency test. 8 In District B, reforms 
included passing an exit test to graduate and mandated remedial 
instruction for students who failed any part of that test. 

Officials and teachers in these districts offered various explanations for 
the benefits accruing to students from these reforms. They associated 
gains with improved instructional practices, curriculum revision and 
alignment, the infusion of basic-skill instruction into general curricula, 
improved teacher training, and greater teacher expectations. We were' 
also told that many teachers were teaching skills measured on state exit 
or national achievement tests used by the districts and that this practice 
had increased because of the increased significance attached to test 
scores. 

While the local informants could have been correct about the intrinsic 
value of remedial programs, those programs were unable to explain our 
results in any straightforward manner, because remediation was imple- 
mented on a different time schedule than the increased student require- 
ments. Remediation was not implemented at all during the first year of 
reform in District C and was only partially implemented in the second 
year, when we measured gains in this district. In District B, more stu- 
dents received remedial instruction during the year before reforms were 



s This requirement has subsequently been extended to a required exit examination, but the compe- 
tency test was in effect during the period through the posttest data reported in this chapter. 
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implemented than during the period between the pre- and posttesting 
for the postreform students. 

The achievement gains may be better explained by the alignment of 
instruction and curricula to test-taking. The comments of officials in 
both districts suggested that tests may be magor factors in instruction 
and curricula. District B officials indicated that many more students had 
failed the relatively difficult mathematics section of the exit examina- 
tion than had failed sections calling for language-arts skills. More dis- 
trict attention, therefore, was directed to teaching mathematics skills 
called for on the exit test, infusing basic mathematics skills into general 
courses, and aligning the mathematics curricula with the exit test. Also, 
District B officials indicated that items on the mathematics section of 
the exit test closely resembled items on the mathematics subtest of the 
standardized achievement test used in the district. This information was 
consistent with the nature of the improvements found in District B; 
namely, the gains were found in mathematics but not in reading. 

The state competency test in District C also seemed to have played a role 
in linking instruction and curricula to the general skills or type of items 
covered in the achievement tests. Reforms there included a state basic- 
skill assessment program under which students were tested at several 
grade levels during their elementary, junior high, and high school 
careers. We were told that curricular, instructional, and promotional 
decisions were closely tied to performance on this test. District officials 
related gains to the comparability of the state competency test and the 
standardized achievement test used in this district. The latter test was 
selected by the state because of its close fit with the state competency 
test. We are unable to distinguish whether increases in scores reflect 
improved mastery in basic skill areas, test items, or both. 

We did not find any information that would explain the absence of gains 
either for Hispanic students or for students not at risk in District B (see 
figures 2.3 and 2.4). 



Districts A and D The reason wh y at-risk students did not benefit in districts A and D as 

they did in the two other districts is not immediately apparent, because 
reforms in all four districts were similar. In order to graduate, postre- 
form students in districts A and D were required to pass an exit test and 
earn additional academic credits. Also, district officials reported similar 
local responses to reforms — improvements in instructional practices, 
greater teacher expectations, and attempts to teach skills measured on 
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exit exams. Why were gains observed in districts B and C and not in 
districts A and D? 

One reason might have been that instruction and curricula seem not to 
have been so greatly founded on exit examinations in districts A and D. 
The exit examinations there were low-risk tests; about 1 percent of the 
graduating class in District D and 2 percent in District A failed to pass 
them. In addition, poor results on these tests were less costly, since they 
were not associated with mandates to provide remedial instruction and 
the subsequent need to hire additional teachers and allocate classroom 
space for remediation. 



Are the Achievement 0ne possibility raised by the explanation of curriculum and instruction 

Gains Real? realignment above is that the gains we reported in districts B and C 

were not educationally meaningful. During our site visits to all four dis- 
tricts, we were told that in the ftce of new testing requirements, some 
teachers were "teaching to the test." "Teaching to the test" could mean 
at least three things: training on exact test items, coaching inexperi- 
enced students on test-taking skills, and focusing instruction on skills 
covered by the test. 

The first meaning— practicing the same questions that will later be 
asked on a specific test— is viewed as having little educational value. 
Higher scores shown by a group with advanced training on exact test 
items compared with an earlier group lacking such practice are not edu- 
cationally meaningful gains. 

The educational value of the second meaning— training possibly inexpe- 
rienced students on test-taking skills— is disputed. Some experts see this 
as more fairly revealing what students have actually learned and level- 
ing the playing field betveen more- and less-experienced students for 
important tests such as those influencing scholarship awards and college 
entry. Ability to do well on standardized or multiple-choice tests can 
influence achievement throughout postsecondary education and at least 
some life opportunities such as entrance into postgraduate training, 
earning required professional licenses and certificates, and some job 
selection. Thus, teaching students how to take tests can be seen as a 
valid aspect of the secondary school curriculum. Differences between 
pre- and postgroups attributable to superior test-taking skills would not 
be interpretable, however, as differences in more generalized 
knowledge. 
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The third meaning — focusing instruction on skills covered by the test — 
quickly gets into complex issues. Where the state or *ocality has care- 
fully identified skills considered essential for students to learn, and then 
has developed fair assessments of whether the students have acquired 
this knowledge, many education experts would agree that focusing 
instruction on these skills represents significant learning opportunities. 
Opinions may differ, of course, on whether the skills originally selected 
have been optim^iy chosen, and how broadly or narrowly the tests may 
assess this Urowledge, but in this context, focusing instruction on skills 
taught by the test is considered educationally appropriate and 
meaningful. 

Thus, the efforts of District C and others to improve the link between 
the curricular and the testing programs are not necessarily negative 
findings if they stem from more-intense, effective teaching of general 
skills, because higher scoies could be interpreted as reflecting better- 
prepared graduates. Similarly, ii is appropriate to modify a curriculum 
to include all the educationally important areas or domains tapped by 
the test. However, improvemr ' ts in test scores compared with those of 
prior groups of students do not necessarily represent greater learning. 
The curricula and instructional practices may have been different but 
educationally sound both before and after reform. The "gains" may sim- 
ply reflect a more appropriate or "fairer" test that taps the postreform 
domains. These issues are sufficiently complex that they cannot be 
resolved with our initial field study. 



Our analysis of these data leads us to three findings and three observa- 
tions Fii st, in our four sites, educational reform did not result in either 
marked losses or substantial gains for educationally at-risk students. 
When improvements occurred, they were modest. While some cases of 
net gain exist, a majority of our comparisons fo»:nd no net gains. Second, 
black students showed a very similar pattern to that of the at-risk stu- 
dents. Third, based on data from three districts, Hispanic students 
showed the least-iavorable trends in reading and mathematics scores. 
Not only were there no net gains for Hispanics from reforms but their 
median scores declined over time in most cases. This was the only group 
we studied that did not appear to share at all in the gains associated 
with reform in some places. 

A first observation based on these findings is that what some had feared 
did not occur: reforms did not provide greater benefits to more- 
advantaged students than to disadvantaged students. 
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A second observation concerns the interpretation of the favorable indi- 
cations of academic progress that we found in three of the eight cases 
and the absence of strong negative findings in any of the remaining five 
cases. Since past evaluations have shown the difficulty in achieving 
even modest gains for educationally disadvantaged students, the mixed 
results reported here should not be dismissed. However, broader conclu- 
sions require data from more school districts. One way of providing this 
information would be through a major national evaluation study of 
school districts with varying experience with education reform. Another 
way would be to accumulate school district studies of the effects of edu- 
cation reforms that cover a range of reforms and time periods. If further 
studies find similar or more favorable effects, evaluators should deter- 
mine whether the test-score gains are educationally meaningful and the 
activities or approaches that seem to be responsible for the positive 
effects. 

The third observation is that further review and study of the type of 
state mandates that force changes in curriculum ani instruction may 
also be warranted. There may be real educational benefits in a program 
that more closely links curricular and instructional content to testing, 
but this may be more likely to occur when test results have imp ortant 
consequence s for districts as well as students . In contrast tn ^strict* a 
and D, districts C and B responded to state mandates that required com- 
pulsory remedial instruction for all students who scored below minimal 
ceilings on state assessment tests. Because results on tests in these dis- 
tricts had a direct effect on the quantity of services that had to be pro- 
vided, district officials were likely to place much emphasis on the 
results of these tests. This appears to be a strong factor in curriculum 
realignment in our two districts that showed achievement gains This 
observation is based on the experience of only four school districts, 
which means that it must be treated more as r- hypothesis than as a 
conclusion. 
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Dropout Rates 



The next questions we asked were whether educationally disadvantaged 
students quit school more in the postreform period under the pressure 
of the new academic requirements and whether they quit school more 
than other students. Two of the four districts provided data that 
allowed us to calculate dropout rates in the prereform and postreform 
cohorts. These districts collected information that enabled us to distin- 
guish students who quit school totally prior to graduation from students 
who left school but subsequently transferred to a private school or a 
school in another district. 

We found no consistent change in dropout rates across the two districts. 
As shown in figure 3.1, the dropout rate for educationally at-risk stu- 
dents across the h*gh school years worsened by 3 percent for the postre- 
form group in District A. It improved by 1 percent in District D. 



Figure 3.1: Dropout Rates for At-Risk 
Students in District* A and D* 
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a Dropout rates were calculated over 3 years of high school in District A and 4 years in District D 
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Generally, dropout rates increased in District A anil declined in District 
D for other groups as well as for at-risk students. Although at-risk stu- 
dents were more likely to drop out than were their higher-achieving 
peers, as shown in figure 3.2, we found dropout rates for students not at 
risk shared the same general pattern shown by at-risk students. We also 
examined dropout rates for minority students. Again, we found the 
same overall pattern as shown in figures 3.3 and 3.4 on pages 43 and 44. 
There was one exception: the dropout rate for Hispanic students in Dis- 
trict A was the same for both cohorts. 
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a Dropout rates were calculated over 3 years of high school in District A and 4 years in District D 



Figure 3.2: Dropout Rates for Students 
Not at Risk in Districts A and D* 
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Figure 3.3: Dropout Rates for Black 
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Figure 3.4: Dropout Rates for Hispanic ■■■HH 
Students in Districts A and D a 

36.0 Psresnt Dropping Out 

33.6 
31.2 




District A District D 

Student Cohort 



] Prereform Cohort 
Postreform Cohort 

a Dropout rates were calculated over 3 years of high school in District A and 4 years in District D 
^ 

Conclusions The educators we s P oke with in the two districts had many views about 

the dropout problem, but we found no consensus that could explain the 
small fluctuations we found in the data. Furthermore, we found no gen- 
eral agreement among school officials as to whether dropout rates, as 
we defined them, were declining or increasing. In both districts, some 
educators thought that dropout-prevention programs and other types of 
extra help were lowering dropout rates, while others thought that 
higher standards were pushing disadvantaged students out of school. 
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The last question we examined was whether educational reforms 
affected educationally disadvantaged students' enrollment in vocational 
education courses. The effect on students' enrollment might be expected 
to come through their schedules: more time spent on new academic 
courses or in remedial work to pass competency tests might have 
reduced opportunities to enroll in elective vocational education courses. 
Low-achieving students in general might have had more trouble meeting 
increased academic requirements and could have been more likely to 
participate in required remedial instruction and to repeat required 
courses. Therefore, reforms could reduce enrollments in vocational 
courses, especially among disadvantaged students. Thus, the questions 
we needed to answer were, first, what the effect of reforms has been on 
vocational education enrollments by at-risk students and, second, 
whether there have been differential effects on at-risk students in com- 
parison to effects experienced by other students. 

To answer the first Ques ion, w^ examined the prereform and postre- 
form educationally disadvantaged students in terms of enrollment in 
academic and vocational courses, based on information from the two 
districts, A and B, that could provide tlv necessary enrollment data. We 
also compared the two groups of disadvantaged students in the type of 
vocational education taken and in the decree of concentrated study in 
vocational education. 

Although aggregated enrollment datr were available from state agencies 
in the four district's states, these data were inadequate to address our 
concerns about the effects of education reform on the enrollment pat- 
terns of at-risk students. The specific limitations of these data were that 
they 

• included all school districts, thus mingling a wide range of variation in 
schedule and vigor of reform; 

• were affected by changes in the size of overall student enrollment, 
which would affect enrollment in all course areas; 

• might reflect variation in practices used by school districts to classify 
courses; 

• included students who had to meet the new requirements and students 
who were under the older requirements for the period of our study; and 

• did not distinguish disadvantaged from nondisadvantaged students. 
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Table 4.1 shows that educationally at-risk, postreform students enrolled 
in more academic courses and fewer vocational education courses than 
did at-risk prereform students. In both districts A and B, the average 
disadvantaged student enrolled in slightly fewer vocational education 
classes — about half a course less — than his or her predecessor. Aca- 
demic enrollment increased, with the average postreform at-risk student 
taking 1 .7 more academic courses in District A and 0.5 more in District 
B. 1 The increase in the number of class periods in District A apparently 
allowed the total number of academic courses to increase without an 
equivalent drop in vocational education enrollment. In District B, the 
drop in vocational education classes was actually greater than the 
increase in academic classes. 



Table 4.1: Average Number of Courses and Change in Number of Courses for At-Rlsk Students Grades 9 Through 11 



Patriot A District B 



Subject area 


Prereform 


Postreform 


Change 


Prereform 


Postreform 


Change 


Vocational 


4.6 


4 1 


-0.5 


4.1 


34 


-0.7 


Consumer, homemaking 


09 


1 2 


0.3 


06 


0.6 


0 


Labor market 


General 


1.3 


08 


-0 5 


09 


09 


0 


Specific 


24 


21 


-0 3 


26 


1 9 


-0 7 


Academic 


23 9 


25 6 


1 7 


145 


150 


05 


Other 


53 


55 


02 


62 


6.1 


-0.1 


Total 


33.8 


35.2 


1.4 


24.8 


24.5 


-0.3 



The declines in vocational education enrollment were in labor market 
preparation courses rather than consumer and homemaking courses. 
Table 4. 1 shows that the average number of labor market preparation 
courses (combining general labor market i^d specific labor market 
courses) taken by at-risk students declined by more than half a course, 
whereas the average number of consumer and homemaking vocational 



Effects on At-Risk 
Students 



The number of courses taken in districts A and B are not directly comparable District A offered 
many more one-semester academic courses than District B, which enabled students in District A to 
enroll in more academic courses This has minimum effect on the /esults we report in this chapter, 
because our statistics report changes between cohorts within the same district 
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education courses taken did not decrease. 2 Declines in labor-market-pre- 
paration course enrollment raise concerns that the educational reforms 
might be indirectly reducing the occupational preparation of disadvan- 
taged students. 

We found a decline in the proportion of at-risk students taking concen- 
trated vocational education programs, defined as rnrollment in five or 
more vocational education courses over 3 years of high school. In Dis- 
trict A, the percentage of disadvantaged students enrolling in five or 
more vocational education courses over 3 years of high school decreased 
by approximately 8 percent. In District B, the percentage of disadvan- 
taged students taking five or more vocational education courses 
decreased by about 16 percent. The additional period of instruction in 
District A might explain the more moderate effect in that district. 



Did the disadvantaged group experience special pressures, because of 
remedial classes or the need to repeat required courses, that might have 
caused a larger drop in total vocational enrollments? Our data from two 
districts demonstrated the same general pattern for disadvantaged stu- 
dents as for nondisadvantaged students: more academic and fewer vo ja- 
tional education courses. 3 For both groups, increases in academic 
courses mostly accrued from added English and mathematics courses, 
whereas declines in vocational education mostly involved decreases in 
labor market preparation courses. (The average number of courses and 
change in number of courses for students not at risk are shown in table 
4.2.) 



2 We used the Secondary School Course Taxonomy, which was developed for the National Assessment 
c° Vocational Education, to identify and classify academic and vocational education courses consist- 
ently across both districts. The classification system divides a high school curriculum into three major 
areas* academic (for example, English, mathematics, and fine arts), personal (health and physical 
education), and vocational education. Vocational education is broken down further into three divi- 
sions One division, consumer and homemaking education, includes courses that prepare students for 
roles outside the paid labor markets, such as consumer economies and family life. A second division, 
general labor market preparation, consists of courses that teach skills that are related to employment 
such as beginning typing, career exploration, and industnal arts A final division, specific labor mar- 
ket preparation, is composed of courses that teach skills that are related to specific occupational 
areas, such as agriculture, business, and communications. 

3 This finding is consistent with the results of other studies such as those conducted by Policy Analy- 
sis for California Education and the Center for Policy Research in Education The former study 
tracked change in course enrollment in California after academic courses had been increased over a 5- 
year period. (The other study is discussed in appendix IV ) Although vocational education sustained 
an overall enrollment decline of 16.79 percent, evidence from the final year demonstrated an increase 
of 8 34 percent, suggesting that the bottom of the decline may have been reached Whether the 
decreases in vocational education courses taken in the districts we studied are the beginning of a long- 
term decline in vocational education enrollment or are likely to be reversed in a few years is a ques- 
tion thai remains to be aaswered. 



Effect on Students Not 
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Table 4.2; Average Number of Courses and Change in Number of Courses for Students Not at Rl»k Grades 9 Through 11 



District A District B 



Subject area 


Prarefonn 


Postreform 


Change 


Pre reform 


Postreform 


Change 


Vocational 


47 


43 


-0.4 


3.5 


3.3 


-0.2 


Consumer, homemakmg 


05 


0.7 


02 


0.4 


0.4 


0 


Labor market 


General 


18 


1 3 


-0 5 


09 


10 


0.1 


Specific 


24 


23 


-0 1 


22 


1 9 


-0.3 


Academic 


288 


29 1 


23 


144 


148 


0.4 


Other 


37 


4.0 


03 


62 


6.2 


0 


Total 


35.2 


37.4 


2.2 


24.1 


24.3 


0.2 



Changes in the types of vocational education courses taken by nondis- 
advantaged students generally paralleled those of the disadvantaged 
group in both districts. The number of labor market preparation courses 
(general and specific) taken by the average student not at risk declined, 
by 0.6 of a course in District A and, to a lesser extent — about 0.2 of a 
course— in District B, while the number of consumer and homemaking 
education courses taken did not decline. 

The proportion of students not at risk taking five or more vocational 
education courses declined as for at-risk students. The extent of this 
change, however, was less dramatic for students not at risk than for the 
at-risk students, particularly in District B. In District A, the proportion 
of students enrolled in five or more vocational education courses 
decreased by 4 percent for students not at risk but by about 7 percent 
for disadvantaged students. In District B, the proportion of students not 
at risk enrolled in five or more courses decreased by 3 percent but by 
about 16 percent for disadvantaged students. The explanation for the 
District B findings may be in the fact that the average number of total 
coui-ses taken increased slightly for students not at risk but declined for 
at-risk students in District B. 



DiSCUSSiOn We found that dec! : nes in vocational education enrollment were associ 

ated with increased academic enrollment in both districts, but the mag- 
nitude and ratio of trade-off differed. For the disadvantaged group in 
District A, the trade-off was a mean loss of half a vocational education 
course per student compared to an increase of 1.7 academic courses per 
student. In District B, the trade-off was more severe: a loss of 0.7 of a 
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vocational education course for 0.5 of an academic course for the disad- 
vantaged group, or a ratio of about two vocational education courses for 
one academic course. 

Reforms affecting students in District A included requirements that 
increased the number of English and mathematics credits needed for 
graduation. The two additional academic courses taken by postreform 
students corresponded to these requirements. Thus, enrollment in aca- 
demic courses could increase without a corresponding decline in voca- 
tional education courses, because the district added one class period per 
day after reforms were implemented. 

In District B, reforms affecting high school students included passing a 
competency test as a graduation requirement and mandatory remedia- 
tion for students scoring below state standards on this test. A substan- 
tial increase in academic courses was not seen in this district, possibly 
because additional academic courses were not required for graduation. 
For disadvantaged students, the small increase in academic require- 
ments that we found might have been associated with mandatory 
remediation, because low-achieving students would be most likely to 
perform poorly on tests and, thus, have to take supplemental remedial 
courses. However, District B provided remediation to low-achieving stu- 
dents before the requirements for competency testing were imple- 
mented. In addition, since the increase in academics was shared by the 
average student not at risk who was not likely to enroll in remedial 
courses, the increase probably reflected factors other than specific high 
school reforms, such as changes in student or parent preference, school 
guidance, course availability, or state college requirements. 

Our data from District A, where increased academic courses were 
required for graduation, suggest that these increases may have affected 
the ability of students to engage in concentrated vocational education 
programs of five or more vocational education courses over 3 years of 
high school. Proportionally fewer postreform students enrolled in five or 
more vocational education courses. Although these requirements may 
not have prevented students from enrolling in some vocational educa- 
tion, they might have had a disparate effect on the subgroup of students 
who chose to enroll in concentrated vocational education programs. 

In both districts, we found that students were taking fewer course that 
taught specific occupational skills. Because most of these courses were 
sequential and were often scheduled for longer blocks of time than other 
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courses, it might have been more difficult for students to find t \iu? to 
schedule these courses and also meet additional academic requirements. 

The small decreases in the average number of vocational education 
courses taken by all students translated into relatively substantial 
deci eases in total vocational education enrollment. In District A, student 
enrollment in vocational education classes declined 7 percent , or about 
2,264 student classes. In District B, overall enrollment in vocational edu- 
cation courses declined 11 percent , or about 610 student classes. Decline 
of these magnitudes may somewhat affect the availability of some voca- 
tional education courses and the number of vocational education teach- 
ers employed. 
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The effects of reform, if any, would be reflected in the difference 
between the actions of students — their achievement performance, drop- 
out behavior, and enrollment patterns — under reform requirements and 
what their actions would have been in the absence of reform. We were 
able to directly measure how the postreform group acted under reform 
requirements, but we had to estimate what they would have done in the 
absence of reform. In our cohort design, we used the achievement scores 
of the prereform cohort as our estimate of what the postreform group 
would have done in the absence of reform. We estimated the effects of 
reform as the difference in actions between the prereform cohort and 
the postreform cohort. 

Any research design tries, with varying degrees of certainty, to control 
factors that may jeopardize the validity of findings. Our main concern 
was with issues of internal validity. That is, Did our inquiry sufficiently 
control for factors other than re *orm that could explain our findings? In 
this section, we review notable factors that might have jeopardized our 
ability to discern the effects of reform and discuss How our design and 
analysis served to control them. 

We would have liked to have stressed external validity (the ability to 
generalize from our findings to school districts other than the ones that 
we studied) as well, but we could not do both in this study. Given our 
resources, our choice of large school districts to allow oversampling of 
at-risk students could not be reconciled with a choice of districts with 
siudent populations that would have been nationally representative. 



If the cohorts experienced different historical events during high school 
besides reform efforts, then differences in achievement, dropout rates, 
or course enrollment might have been caused by these other events 
rather than by reform. One of the strengths of our cohort design is that 
the ac^acent cohorts going through high school shared an overlapping 
history. Having entered high school only 1 year apart (2 years apart in 
District A because of a reform transition year), the two cohorts' largely 
overlapping histories made it less likely that events other thrs. reform, 
such as charges in job opportunities for dropouts, might have explained 
differences in their actions. 

However, since reform history also overlapped the experiences of both 
cohorts, reform might have indirectly affected the actions of the prere- 
form cohort. If this occurred — for example, if reform induced changes in 
course offerings, the enrollment patterns of the prereform cohort might 



The Cohort Design 



Experience During the 
Years Studied 
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have been affected also — then we would underestimate the effects of 
reform. We knew, however, that the direct effects of additional gradua- 
tion requirements mandated by reform did not apply to the p^ereform 
cohort. 1 



Initial Cohort Differences If the tw0 st ^dent groups differed in ways significant to outcomes, such 

as higher median achievement among at-risk students in one or the 
other cohort, later differences might have resulted from these initial dif- 
ferences rather than from reform. A truly experimental approach to 
controlling initial differences would have been to randomly assign stu- 
dents to reform and not-reform experiences, but this option was obvi- 
ously not possible for after-the-fact studies such as ours. 



Even if the compared cohorts did not have initial differences in the 
eighth grade, differential losses of students from the two cohorts during 
high school might also confound our findings about the effect of reform 
on student achievement. For example, if poorer-achieving students were 
more Jikely to drop out of the postreform group, then chis group's supe- 
rior performance in the high school test might have resulted from hav- 
ing a larger percentage of better students remaining in school rather 
than from students' learning better. 

In our analyses of achievement and enrollment, we limited our analyses 
to "survivors" to avoid contour ding our results with differential loss of 
students. We analyzed only the students for whom the school had the 
necessary data on achievement and enrollment over the years we 
examined. (Note that the necessary data might not be available for a 
number of reasons, including withdrawal from school, not taking a test 
for whatever reason, or failure of the district to enter data.) While sur- 
vivor analysis limited our findings to students for whom data were 
available, it ensured that within-cohort changes in median achievement 
were not caused by differences in attrition rates. Also, survivor analysis 
allowed us to examine the differences in average course enrollment over 
the same number of years, because students who did not attend school 
for the entire period we studied, and who consequently enrolled in fewer 



^me prerefoim students might have been directly affected if they were required to meet reform 
requirements because they failed to progress normally after entering the ninth grade (their first year 
of high school), thus graduating after reforms %vere implemented. As noted in chapter 1, these mixed- 
treatment cases did not significantly affect our analysis, because m two districts they v. ere a small 
percentage (less than 10 percent) m our achievement analysis, and, in the two other districts, pre re- 
form students who were retained were allowed to graduate under the older requirements. 



Cohort Differences Over 
Time 
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courses, were excluded from our analyses. However, the use of survivor 
analysio obscured differential losses of nonsurvivors between education- 
ally at-risk students and students who were not at risk. 

In the case of our achievement analyses, our data on eighth-grade scores 
suggested that the combined factors of initial diffe r ences in achievement 
and subsequent differences in student loss were not important problems. 
In three of our districts, the analyzed prereform and postr form groups 
had quite similar eighth-grade scores. (Note in figure 2.1 that the initial 
median scores between the groups tended to be close or identical.) In the 
remaining district, differences in ei£ith-grade scores were exacerbated 
by the transitional year that separated the cohorts in only this district. 
Furtheriiiore, our use of gain-score analysis, wliich compared changes in 
scores between eighth grade and high school rather than differences in 
high school scores, served to adjust for initial differences. (This ac^just- 
ment, however, could not resolve any bias arising from anything other 
than tl e reform, which might have produced greater change in the 
mediae bcores in one cohort than another.) 



Regression Toward the 
Mean 



If groups are selected on the basis of extreme scores, there is a tendency 
for a group's score to move closer to the mean of the overall group upon 
retesting. As a result of this statistical tendency, called "regression 
toward the mean," the test scores of a group selected as low achievers 
would be expected to improve upon retesting, even if reform had no 
effect at all. We have avoided confounding regression toward the mean 
with the effect of reform by comparing the differential gains between 
two identically defined extreme groups, the prereform and postreform 
groups of at-risk students. Because we used the same criterion to define 
at-risk groups in the prereform and postreform cohorts, we have no rea- 
son to believe that this statistical tendency influenced scores of the at- 
risk groups differentially. We have cautioned against making compari- 
sons between the postreform groups of students at risk and students not 
at risk, because regression toward the mean is likely to result in 
observed improvements among at-risk students but not among postre- 
form students not at risk, independently of the effects of reform. 
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Measurement Issues 



Achievement Data We used scores on norm-referenced achievement tests to measure 

achievement, because they provided a standard and reliable measure to 
compare the differences between eighth-grade scores and high school 
scores for students in the prereform and postreform cohorts. Although 
course grades were available in some districts, we thought that course 
grades were too subjective to allow meaningful comparisons. We also 
rejected the use of Scholastic Aptitude Test (SAT) and American College 
Testing (ACT) Program examination scores. Since the SAT and ACT are 
usually taken only by college-bound students, scores on these tests 
would not be available for large numbers of at-risk students, * > were 
the focus of our study. Furthermore, scores on these tests are uiu itable 
for measuring relative gains over time, since they are taken only toward 
the end of students' high school careers, and no comparable test is given 
earlier. 

When a school district changes from an earlier to a more recent edition 
of an achievement test, average scores might appear to increase or 
decrease substantially as a result of changes in the norm group or test 
item from one edition to the other. This phenomenon did not affect our 
analyses. In two of the districts, both cohorts were tested on two occa- 
sions with the same edition of the test. In the two other districts, new 
editions of the test were introduced during the years we examined. How- 
ever, we were able to use scores that had been equated from the newer 
edition to the older edition, so that the scores we compared were 
equivalent. 

We analyzed changes in achievement by comparing the differences 
between the median percentile scores obtained by prereform and postre- 
form group on eighth-grade tests with those obtained on tests taken dur- 
ing the third year of high school. 2 We used percentile scores because 
percentiles were the only common measure available across all four dis- 
tricts. Because percentiles are unequal units that oo not conform to a 
normal curve, analyses comparing differences in outcome for the median 
subject were most suitable for our data. 



J In District C, high school achievement tests were routinely administered to only iOth-grade students 
1 herefore, we used IOth-grade scores for our posttest comparison 
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We were unable to control for district or school practices that might 
have excluded certain students from testing. While information from 
one district indicated that students in regular high school placements 
who were handicapped or suspected of being handicapped might have 
been excluded from testing, information from the districts did not sug- 
gest that large groups of students were exempt from testing. Local offi- 
cials indicated that district testing practices did not change when 
reforms were introduced. If district or school practices might have 
excluded some students f om testing, we have no reason to believe these 
practices had differential effects on outcomes for the prereform and 
postreform cohorts. 



Dropout Rates ^ w0 districts were able to supply attendance data that included reasons 

for withdrawal from school. We computed dropout rate:, by tracking all 
ninth-grade students in the prereform and postreform groups in these 
districts over their high school years. We tracked students in District A 
through the beginning of their third year of high school and students in 
District D through the beginning of their fourth year cf high school. We 
defined dropouts as students who left school without graduating or 
transferring to a private or out-of-district school. 1 This method allowed 
us to control for the effects of residential migration into and out of a 
district. 

We chose to compute dropout rates rather than use rates provided by 
the districts because of the limitations these data presented. In one dis- 
trict, foi example, dropout rates were computed while ignoring with- 
drawal codes that indicated students enrolled in schools elsewhere. In 
two districts, dropout data were computed for the school year and did 
not include students who dropped out during the summer. In addition, 
district data did not control for the effects of students moving into the 
district, nor did they allow us to distinguish differences between rates 
for at-risk students and students not at risk or to track the rates for 
prereform and postreform cohorts. 

Despite the obvious strengths of our method, our data also presented 
some limitations. First, we were unable to verify tho withdrawal codes 
provided by the districts and, therefore, we were unable to know if 



^Students who indicated that they planned to enter a general equivalency diploma (GED) program 
were counted a.< dropouts. 
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codes were used reliably or consistently from one year to another. Sec- 
ond, we were unable to identify students who dropped out but who sub- 
sequently reenrolled in school. A third limitation was that students who 
dropped out of school before entering the ninth grade were not included 
in our analyses. Although this was not technically a limitation, because 
our study focused on the effects of reform on high school students, the 
omission of seventh- and eighth-grade dropouts reduces the comparabil- 
ity of our results with those of studies that included these students. 
Finally, our data did not record dropping-out behavior over the total 
time students were in high school. 
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In this appendix, we present the data for chapter 2. The table numbers 
correspond to figure numbers in chapter 2. 



Table 11.1: At-Risk Students 



Cohort 


Median percentile score 
Grade 8 High school 


Net gain 1 


Mathematics 


District A 


Prereform 


10 


21 




Postreform 


20 


27 


-4 


District B 


Prereform 


35 


31 




Postreform 


35 


39 


8 


District C 


Prereform 


27 


24 




Postreform 


27 


31 


7 


District D 


Prereform 


32 


34 




Postreform 


33 


34 


-1 


Reading 


District A 


Prereform 


8 


20 




Postreform 


13 


22 


-3 


District B 


Prereform 


23 


17 




Postreform 


22 


15 


-1 


District C 


Prereform 


2' 


16 




Postreform 


20 5 


20 5 


5 


District D 


Prereform 


15 


19 




Postreform 


17 


18 


-3 



in scores for the postreform cohort 
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Table 11.2: Black Students 



Cohort 




Median oercentile score 






Grade 8 


High school 


Net gain" 


Mathematics 


District A 


Prereform 




13 


27 




Postrefoim 




31 


33 


-12 


District B 


Prereform 




48 


41 




Postreform 




47 


47 


7 


District C 


Prereform 




35 


33 




Postreform 




35 


41 


8 


District D 


Prereform 




36 


36 




Postreform 




40 


39 


-1 


Reading 


District A 


Prereform 




15 


25 




Postreform 




23 


29 


-4 


District B 


Prereform 




36 


27 




Postreform 




36 


26 


-1 


District C 


Prereform 




32 


24 




Postreform 




31 


33 


10 


District D 










Prereform 




25 


23 




Postreform 




29 


24 


-3 



a Net gain was computed by subtracting the change in scores lor ;he prereform cohort from the 
in scores lor the postreform cohoit 
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Table 11.3: Hispanic Students 



Cohort 


Median percentile score 
Grade 8 High school 


Net gain* 


Mathematics 








District A 


Prereform 


21 


33 




Postreform 


41 


43 


-10 


District B 


Prereform 


46 


42 




Postreform 


51 


47 


0 


District C 


Prereform 


b 


b 




Postreform 


b 


b 


t 


District D 


Prereform 


63 


59 





Postreform 



60 



56 



Reading 



District A 


Prereform 


20 


28 




Postreform 


32 


38 


-2 


District B 


Prereform 


35 


26 




Postreform 


39 


25 


-5 


District C 


Prereform 


b 


b 




Postreform 


b 


b 


t 


District D 


Prereform 


46 


43 




Postreform 


46 


41 


-2 



a Net gam was computed by subtracting the change in scores for the prereform cohort from the change 
in scores for the postreform cohort 
b Not applicable 
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Table 11.4: Students Not at Risk 





Median percentile score 




Cohort 


Grade 8 


High school 


Net gain* 


Mathematics 


District A 


Prereform 


63 


59 




Postreform 


64 


61 


1 


District B 


Prereform 


61 


55 




Postreform 


63 


55 


-2 


District C 


Prereform 


57 


61 




Postreform 


61 


70 


5 


District D 


Prereform 


/ 0 


73 





Postreform 



76 



73 



Reading 



District A 


Prereform 


60 


65 




Postreform 


58 


62 


-1 


District B 


Prereform 


51 


42 




Postreform 


53 


41 


-3 


District C 


Praeform 


64 


58 




Postreform 


64 


68 


10 


District D 


Prereform 


68 


62 




Postreform 


69 


60 


-3 



a Net gam was computed by subtracting the change in scores for the prereform cohort from the change 
in scores for the postreform cohort 
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Educational reforms might have had different effects in rural than in 
urban districts. 1 Difficulties in implementing reforms and disruptions 
associated with reforms might have been greater in rural areas than in 
urban areas because rural districts have fewer resources. Also, reforms 
in rural areas might have done less to improve achievement or dropout 
rates of disadvantaged students, because resources needed to provide 
appropriate interventions for disadvantaged students might have been 
less available. Although a great deal of divergence exists among rural 
areas, rural areas generally have lower tax bases, because they have 
relatively fewer inhabitants, and a disproportionate number of rural 
inhabitants are poor. Schools in rural areas tend to be smaller and pro- 
vide fewer course offerings. In addition, rural districts often have 
greater difficulty in recruiting and retaining teachers. 

Additional graduation requirements and compulsory remedial courses 
might have had a greater effect on vocational education enrollment in 
rural schools than in nonrural schools. The need to direct additional stu- 
dents, teachers, and funds to academic areas might divert resources 
from vocational areas. This redirection might have had serious conse- 
quences on the availability of vocational courses and on the ability of 
students to enroll in vocational courses, because rural schools tend to 
lack sufficient students to support diverse program offerings. Thus, per- 
sonnel and economic constraints might be expected to impose additional 
limitations on the ability of schools to provide diversified vocational 
education program offenngs. 

In order to investigate the effect of educational reforms in rural areas, 
we conducted an opinion survey with approximately 20 rural school 
officials and researchers with direct knowledge of the effects of educa- 
tional reforms in rural areas. Among the individuals we surveyed, we 
included officials from two school districts, one from a high-poverty 
area and one from a low-poverty area, in the four states where we stud- 
ied the effects of reforms in urban districts. 

This approach allowed us to collect information from a sizable number 
of informants with knowledge of the effects of reform on diverse rural 
districts, so that we might compare the effects of reform in rural and 
urban districts. Since the survey was not based on a random sample, we 
cannot generalize our findings to other rural districts. We did not 



'We were unable to include rural districts in our study Our analyses required that we select districts 
that served large numbers of disadvantaged students and maintained historic student data on com- 
puter tapes. We were unable to locate rural districts that satisfied both these criteria. 
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attempt to validate the data or observations underlying the opinions we 
solicited. 



Implementation of 
Educational Reforms 



We found the resources to implement reforms varied widely among 
rural districts, including rural districts in the same state. Several respon- 
dents indicated that rural districts were forerunners in implementing 
reforms. These districts had the resources to implement reforms that 
met or exceeded state reforms that were passed later. However, other 
respondents indicated that the implementation of reforms was burden- 
some and stretched the financial and administrative capacities of rural 
schools. For example, a rural school official mentioned that guidance 
counselors in his district were functioning as record keepers rather than 
counselors. The tracking requirements imposed by reforms were so cum- 
bersome that counselors no longer had time to provide counseling ser- 
vices to students. 



Educational reforms appeared to have both positive and negative 
effects on educational outcomes for educationally at-risk students. Rural 
officials, like their urban counterparts in the districts we visited, 
reported that many educationally disadvantaged students were experi- 
encing gains in achievement because of the availability of remedial 
instruction and other special programs targeted toward the disadvan- 
taged that were funded as part of educational reform in many states. In 
addition, respondents indicated that new state mandates were useful in 
leveraging additional funds to support district-level programs from local 
school boards. 

According to some respondents, student frustration increased as a result 
of increased academic course requirements. Students who had failed any 
courses (particularly those who attended schools offering fewer than 
seven periods) experienced much difficulty in scheduling the expanded 
requirements. We were told that many of these students elected to drop 
out rather than remain in school an additional year. 

"No pass, no play" rules that prevented students who failed one or more 
courses from participating in extracurricular activities might have had a 
greater effect on student frustration in rural areas than in nonrural 
areas. Perhaps because of the importance of extracurricular activities to 
the social lives of rural students, respondents associated banmg stu- 
dents from these activities with increased frustration and increased 
dropout rates. 



Effects of Reform on 
Achievement and 
Dropout Rates 
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While most respondents reported increased student frustration and 
higher dropout rates during the early years of reform, others indicated 
that dropout rates were stable or declining. In districts where declining 
dropout rates were reported, improvements were attributed to postre- 
form dropout prevention programs and remedial supports, 



Slightly more than half the respondents indicated that vocational educa- 
tion enrollment was declining in rural districts. As in the urban districts 
we visited, decline was associated with increased academic graduation 
requirements, mandated remedial instruction, and the redirection of 
resources from vocational areas to academic areas. Respondents told us 
that additional required courses might have reduced the number of 
vocational education courses under some conditions. Mandates that 
increased the number of academic or remedial courses that students 
must take tended to decrease the number of elective courses, including 
vocational education. In contrast, some districts added an extra period 
that allowed students to take more academic courses without a corre- 
sponding decrease in vocational education courses. Other districts his- 
torically offered relatively long eight-period instructional days, which 
made it possible for students to meet new requirements and still have 
opportunities to enroll in vocational education courses. 

Several respondents indicated that rural districts had difficulties 
recruiting mathematics and science teachers, and these difficulties were 
exacerbated when reforms increased the demand for teachers certified 
in this area. Many districts granted emergency certifications to voca- 
tional education teachers and teachers of other elective subjects to fill 
newly created academic positions. In these districts, fewer teachers were 
assigned to vocational education courses after reform, which might have 
limited the availability of vocational courses. 

The lack of adequate ph/sical facilities and classroom space also 
appeared to limit the availability of vocational education courses. Some 
respondents said that reforms that required reductions in class size and 
supplemental remedial classes taxed the physical capacity of some rural 
schools and made it necessary to reassign vocational education class- 
100ms to academic classrooms. 



Effects of Reform on 
Vocational Education 
Enrollment 



Conclusion Informs reportedly have had divergent effects on schools in rural areas. 

We were told that the academic level and graduation rates of disadvan- 
taged students improved when resources were adequate and remedial 
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The Effects of Reform in Rural 
School Districts 



and other student supports were in pi -\ We were also told that when 
the school day was sufficiently long anu vocational education course 
offerings were readily available, vocational education enrollment did not 
decline. However, reforms appeared to be placing a disproportionate 
burden on the administrative capacities of rural schools and on the abil- 
ity of rural schools to provide diverse course offerings. 
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Related Multistate Studies 



This is the first multistate study to assess the effects of reform on the 
achievement, dropout rates, and course enrollments of disadvantaged 
high school students. We searched for related multistate studies, using 
both computer data base searches and phone interviews with experts in 
the field. With regard to academic achievement, we were unable to iden- 
tify any other multistate study that assessed the effects of educational 
reforms upon the achievement of high school students. However, a mul- 
tistate study examined the effects of reform on dropout rates. Another 
multistate study explored the relationship of refoims to course offerings 
available to high school students while another study compared student 
course enrollment patterns over a time span during which many states 
implemented reforms. We cite these four studies and summarize their 
findings below. 



Effects of Reforms on 
Dropout Rates 



Amelia E. Kreitzer, George Madaus, and Walt Haney explored the rela- 
tionship between state minimal competency testing requirements and 
dropout rates in the 10 states with the highest and lowest dropout rates 
in 1986. 1 Of the 10 states with the lowest dropout rates, five had no 
competency testing requirements and none had requirements that stu- 
dents must pass an exit test to graduate. Of the 10 states with the high- 
est dropout rates, all had competency testing requirements and nine of 
the 10 used tests to determine eligibility for high school graduation. In 
explaining this apparent link between dropping out and minimal compe- 
tency tests, these authors concluded that although state minimal compe- 
tency requirements might have increased the dropout rates for students 
at risk of dropping out in situations where graduation and promotional 
decisions were based on results from these tests, it is more likely that 
high dropout rates were among the conditions that motivated state legis- 
latures to mandate minimal competency test requiremonts in the first 
place. Although this study did not establish a causal relationship 
between minimal competency testing and dropout rates, it raised impor- 
tant questions regarding the relationship between reforms that 
increased academic requirements and changes in dropout rates. 



Enrollment Patterns 



The Center for Policy Reseerch in Education examined the effects of 
increased iJgh school graduation requirements on course offerings in 



'Amelia E. Kreitzer, George F. Madaus, and Walt Haney, "Competency Testing and Dropouts," forth- 
coming. The authors used state-level data from the U S. Department of Education. The limitations of 
using aggregated data were noted earlier in this report. 
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four states where graduation requirements had been increased. 2 Using 
data provided by a nonrandom sample of 19 high schools in 13 districts, 
the center found that the most common changes in course offerings were 
additions of mathematics and science courses: 12 of the 13 districts 
reported adding sections in these areas. 3 The sections added were 
predominantly at the basic level. Of the 17 schools adding mathematics 
sections, 15 reported adding basic, general, and remedial sections. Of the 
16 adding science sections, 14 reported adding basic, remedial, or gen- 
eral-level course offerings. Reductions were reported in elective courses 
such as vocational education, physical education, and performing arts 
and were attributed to the new graduation requirements. Because of dif- 
ferences in methodology and data, we were unable to directly compare 
these results with our findings. However, these results were consistent 
with our finding that students in districts where graduation require- 
ments increased were taking more academic courses and fewer voca- 
tional courses. 

The National Assessment of Vocational Education compared changes in 
course enrollments between 1982 and 1987. 4 From data from student 
transcripts, it was found that graduates of the high school class of 1987 
took an average of nearly two more credits than the graduates of the 
high school class of 1982. (A credit was defined as a typical course that 
meets five periods per week.) It was also found that students who grad- 
uated in 1987 enrolled in slightly less vocational education than stu- 
dents who graduated in 1982. Vocational education enrollment declined 
slightly from an average of 4.38 credits in 1982 to 4.24 credits in 1987. 
The authors suggested that declines in vocational education enrollment 
might be larger in future years as more states implement reforms that 
increase academic graduation requirements. 

Findings from another study suggested that increased graduation 
requirements were restricting vocational education enrollment in some 



2 Wilham Clune, Paula White, and Janice Patterson, The Implementation and Effects of High School 
Graduation Requirements: First Steps Toward Curncular Reform (New Brunswick, New Jersey: 
Center for Policy Research in Education, 1989). 

3 The data were collected during interviews dunng the spring of 1987 In three states, increases in 
requirements were implemented in that year. In the fourth state, reforms were not scheduled to be 
implemented until 2 years later However, school officials reported that the necessary courses had 
been put in place so that students could meet the new requirements before their senior year 

4 National Assessment of Vocational Education, testimony of John G Wirt before tir ? Subcommittee on 
Elementary, Secondary, and Vocational Education, Committee on Education and Labor, U.S. House of 
Representatives, March 7, 1989. Although many state reforms were implemented between 1982 and 
1987, the national random sample included data from states that did not pass reforms that increased 
graduation requirements as well as from states that had not yet implemented new requirements. 
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states. Using aggregated enrollment data obtained from qu?stionaires 
sent to state directors of vocational education, Nevin R. Frantz, Deborah 
C. Strickland, and Donald E. Elson found that vocational enrollment*; in 
over half the states had either declined at a faster rate than overall high 
school enrollment or had increased at. a slower rate than the number of 
high school students between the 1982-83 and 1986-87 schooi year. 5 
These authors concluded that state-level actions, such as requiring a 
vocational education course or providing funding incentives for voca- 
tional education, appeared effective in preserving vocational education 
programs. 



R Nevtn R FranU Jr , Deborah C Stnctland, and Donald K Bison, "Is SeconH«n' Suucatior at Risk?" 
Vocational Education Journal , O^*ober 1988, pp 34-37 
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